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Abstract 

A critical aspect of intervention research is investigating the active ingredients that underlie intensive interventions and 
their theories of change. This study explored the rate of instructional interactions within treatment groups to determine 
whether they offered explanatory power of an empirically validated Tier 2 kindergarten mathematics intervention. Direct 
observations were conducted in 46 interventions groups, involving approximately 228 students. Multilevel structural 
equation models revealed that fall mathematics achievement significantly and negatively predicted the rate of academic 
feedback. Specifically, intervention groups with lower student mathematics achievement at the start of the kindergarten 
year received higher rates of academic feedback. Analyses also suggested that latent rates of instructional interactions 
were not significantly correlated with gains on student mathematics outcomes. Implications are discussed in relation to 
specifying the underlying mechanisms to intensify mathematics interventions, documenting local adaptation of intervention 
implementation, and examining possible threshold effects of instructional interactions. 
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The research agendas of prominent, federal funding agen- 
cies, such as the Institute of Education Sciences (IES) and the 
National Science Foundation, have led to scientific break- 
throughs in the field of educational research. At the forefront 
of many of these advancements have been methodologically 
sophisticated efficacy trials (e.g., randomized controlled tri- 
als), where the primary purpose is to produce high-quality, 
trustworthy evidence about the impact of an intervention on 
targeted outcomes relative to a counterfactual or a compari- 
son condition. Efficacy data are an integral component for 
building the knowledge base on effective instructional prac- 
tices and educational tools. However, alone these data are 
insufficient to draw reliable inferences as to why an interven- 
tion works, for whom, and under what conditions. 

To better understand the evidence behind an interven- 
tion’s impact on student outcomes, researchers need to 
investigate the active ingredients that underlie interventions 
and their theories of change. Active ingredients represent the 
core mechanisms through which mathematics interventions 
operate. Investigating the active ingredients of interventions 
may increase researchers’ capacity to explicate how or under 
what conditions interventions produce positive or, in some 
cases, negative treatment effects. The purpose of this article 
was to explore the theoretically specified mechanisms that 


comprise an empirically validated Tier 2 mathematics inter- 
vention to determine whether they had explanatory power 
contributing to the improvement of student mathematics 
outcomes within the treatment condition. 

Effects of Systematic and Explicit 
Mathematics Interventions 

Much of what the field knows about instruction for students 
with or at risk for mathematics difficulties (MD) has been 
derived from studies that examined interventions with 
explicit and systematic instructional foundations (Gersten 
et al., 2009). In many cases, the impact of these interven- 
tions on student mathematics outcomes was contrasted with 
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a business-as-usual control condition. For example, L. S. 
Fuchs et al. (2005) investigated the efficacy of preventive 
tutoring on the mathematics outcomes of 139 first-grade 
students with MD. Participating at-risk students (AR) were 
randomly assigned to a tutoring condition or a control con- 
dition that provided core mathematics only. AR students 
assigned to the tutoring condition received 30 min of small- 
group tutoring and 10 min of technology-based practice to 
build math facts fluency. Small-group tutoring utilized 
explicit mathematics instruction to teach key concepts and 
skills associated with whole numbers. Results significantly 
favored the AR students assigned to the tutoring condition 
over their AR control peers on standardized measures of 
whole number computation, problem solving, and whole 
number concepts and application, with effect sizes 
(Hedges’s g ) ranging from .14 to .70. 

More recently, Bryant et al. (2011) conducted a random- 
ized controlled study to examine the effects of a systematic 
and explicit Tier 2 mathematics intervention on the mathe- 
matics achievement of first-grade students with MD. 
Approximately 200 students were randomly assigned to a 
business-as-usual control condition or the treatment inter- 
vention, which focused on early numeracy concepts and 
skills. Statistically significant treatment effects were found 
on proximal measures of whole number computation and 
aspects of number proficiency. Bryant and colleagues 
reported effect sizes (Hedges’s g) ranging from .39 to .55. 

In our own work, we have studied the efficacy of the 
ROOTS intervention, a Tier 2 kindergarten program that 
was engineered to help kindergarten students with or at risk 
for MD develop a deep understanding of whole numbers. 
Central to the ROOTS intervention’s capacity to meet the 
instructional needs of students with MD is an “instructional 
platform” (Simmons, 2015) that carefully integrates foun- 
dational concepts of whole number and validated principles 
of systematic and explicit mathematics instruction (Coyne, 
Kame’enui, & Camine, 2011; Gersten et al., 2009). Recent 
efficacy trials funded by IES have documented the interven- 
tion’s positive impact on important student mathematics 
outcomes (Clarke et al., 2016; Clarke et al., in press). 

Our initial study of the ROOTS intervention involved 
29 classrooms that were part of a larger randomized con- 
trolled trial (Clarke et al., 2016). Fourteen classrooms 
were in the treatment condition, and 1 5 were in the control 
condition. Teachers in both conditions were asked to nom- 
inate the five lowest performing students. A total of 140 
students were determined eligible for ROOTS, with 67 
students in treatment classrooms and 73 students in con- 
trol classrooms. Students in the treatment classrooms 
received ROOTS in small-group formats (i.e., three-five 
students), 3 days per week for approximately 18 weeks. 
The intervention was delivered by school-employed para- 
professionals, who had experience working with students 
with learning difficulties. Control students received core 


mathematics instruction only. Nested time by condition 
analyses suggested that ROOTS students made stronger 
gains than their control peers across kindergarten on a 
standardized outcome measure of mathematics (Hedges’s 
g = .38) and a set of four early numeracy curriculum-based 
measures (Hedges’s g = .30). 

The most recent study of ROOTS utilized a randomized 
block design (Clarke et al., in press), with 290 students ran- 
domly assigned within classroom to one of three conditions: 
(a) a ROOTS intervention group with a 2:1 student-tutor 
ratio (n = 58), (b) a ROOTS intervention group with a 5:1 
student-tutor ratio (» = 145), or (c) a no-treatment control 
condition ( n = 87). Students in the ROOTS groups continued 
to receive district-approved core mathematics instruction. 
Control students, however, received district-approved core 
mathematics instruction only. Results showed that ROOTS 
students demonstrated greater mathematics achievement 
gains than their peers in the control condition on four of the 
six outcome measures. Specifically, reported effect sizes 
(Hedges’s g) ranged from .28 to .74 (Clarke et al., in press). 

It appears from our studies and other efficacy studies of 
early mathematics interventions (e.g., Bryant et al., 2008; L. 
S. Fuchs, Fuchs, & Compton, 2012; Sood & Jitendra, 2013) 
that systematically designed and explicitly delivered math- 
ematics interventions are critical for students with or at risk 
for MD. In the case of ROOTS, the evidence regarding the 
intervention’s impact on student mathematics achievement 
was generated in efficacy trials that focused primarily on 
comparisons between the treatment and control conditions 
(Clarke et al., 2016; Clarke et al., in press). Consequently, 
much is still to learn about the underlying mechanisms 
behind ROOTS and its theory of change. 

The Theory of Change for the ROOTS 
Intervention 

In the fields of curriculum development and intervention 
research, a well-specified theory of change represents the 
underlying mechanisms of an intervention that are hypoth- 
esized to increase student achievement when implemented 
as intended (Doabler, Clarke, et al., 2015). Figure 1 depicts 
the theory of change for the ROOTS intervention. As 
depicted, the intervention involves three key tenets: (a) 
intervention components, (b) high-quality instructional 
interactions, and (c) proximal and distal student outcomes. 
Two intervention components comprise the ROOTS pro- 
gram: (a) whole number concepts and skills, and (b) vali- 
dated principles of systematic and explicit mathematics 
instructions. When systematically integrated, these two 
components are anticipated to facilitate high-quality 
instructional interactions between teachers and students, 
and among students, around critical concepts and skills of 
whole numbers. Such interactions are hypothesized to 
mediate the effects of ROOTS on proximal and distal 
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Figure I. ROOTS theory of change. 


mathematics outcomes. Proximal mathematics outcomes 
include students’ conceptual understanding of key con- 
cepts and procedural fluency with mathematical problems. 
In Figure 1, student mathematics achievement is depicted 
as the distal outcome. 

Mathematical Content of the ROOTS 
Intervention 

ROOTS is a 50-lesson, Tier 2 kindergarten intervention 
program that was designed to support students in develop- 
ing early mathematics proficiency (Clarke et al., 2016). 
Each 20-min lesson consists of four to five mathematics 
activities that center on three domains of whole number 
understanding identified in the Common Core State 
Standards (CCSS) Initiative (2010): (a) Counting and 
Cardinality (b) Operations and Algebraic Thinking, and (c) 
Number and Operations in Base Ten. School-based person- 
nel deliver the lessons in small-group fonnats, with two to 
five students per group. 

Because at-risk kindergarten students tend to experience 
difficulties understanding and working with concepts and 
skills of whole number, the first intervention component of 
the ROOTS intervention invariably prioritizes developing 


students’ early number sense. Number sense refers to a 
child’s awareness of and fluidity with numbers (Berch, 
2005; Gersten & Chard, 1999). As identified in the CCSS 
Initiative (2010) and recommended by experts (Gersten 
et al., 2009; National Research Council [NRC], 2001), 
foundational aspects of number sense addressed in the 
ROOTS intervention include using efficient counting strat- 
egies (e.g., min strategy), recalling answers to basic number 
combinations, such as 5 + 2, gaining foundations of the 
base-ten system, and solving simple addition and subtrac- 
tion word problems. 

Principles of Systematic and Explicit 
Instruction 

The second intervention component of the ROOTS interven- 
tion attends to the instructional design and delivery principles 
that form the architectural foundation of the intervention. 
Converging evidence suggests that students with MD signifi- 
cantly benefit from instruction that is systematically designed, 
explicitly delivered, and includes a purposive selection of 
scaffolded student learning opportunities and visual repre- 
sentations of mathematical ideas (Gersten et al., 2009; 
National Mathematics Advisory Panel [NMAP], 2008). 
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The ROOTS intervention was developed with specific atten- 
tion to the systematic and explicit instructional design and 
delivery principles that have been validated to improve the 
mathematics outcomes of at-risk learners (Gersten et al., 
2009). These principles include (a) engaging students’ prior 
understandings of mathematics, (b) providing vivid demon- 
strations and clear explanations of mathematical concepts, 
(c) using visual representations of mathematical ideas to pro- 
mote conceptual understanding, (d) providing opportunities 
for practice and review to promote mathematical proficiency, 
and (e) delivering timely academic feedback to address stu- 
dent misconceptions. 

Instructional Interactions in Early 
Mathematics 

Instructional interactions are often viewed as a critical ele- 
ment of academic development (NRC, 2001; Pianta & 
Hamre, 2009) and a defining feature of systematic and 
explicit interventions (Gersten et ah, 2009). Systematic and 
explicit mathematics interventions are designed to facilitate 
high rates of important instructional interactions between 
teachers and students around critical whole number con- 
cepts. Such instructional interactions represent a dynamic 
interplay of the teacher offering clear examples and demon- 
strations of new mathematics content, students working 
independently and with their peers in structured learning 
opportunities, and the teacher providing specific, informa- 
tional feedback in response to student learning (Doabler, 
Baker, et ah, 2015). For example, a systematic and explicit 
intervention will support teachers in facilitating structured 
opportunities for students to verbalize their mathematical 
thinking and understanding. 

The ROOTS intervention facilitates three types of sys- 
tematic and explicit instructional interactions: (a) overt 
teacher models, (b) deliberate student practice opportuni- 
ties, and (c) specific academic feedback. For example, the 
intervention offers tutors scripted guidelines to overtly 
explain and demonstrate critical concepts of whole num- 
ber. The intervention also assists tutors in facilitating fre- 
quent student practice opportunities, such as allowing 
students to verbalize how they solved a mathematics prob- 
lem and use place value blocks to represent teen numbers. 
In addition, ROOTS offers strategies for providing aca- 
demic feedback to address student errors and affirm correct 
responses. 

Each ROOTS lesson contains between 100 and 150 pre- 
scribed instructional interactions. Such interactions vary in 
terms of how they are detailed in the program. Some inter- 
actions, for example, are fully scripted, providing tutors 
specific directions on how to pose mathematical questions 
and demonstrate mathematics concepts. In other cases, the 
intervention offers reminders for tutors to facilitate instruc- 
tional interactions. For example, after a mathematical 


concept or skill is introduced, the intervention scripting 
makes recommendations for tutors to “provide individual 
turns,” “repeat with other pairs of children,” “confirm or 
provide corrective feedback,” or “repeat with the next 
number sentence.” 

To date, we have yet to explore the nature of the instruc- 
tional interactions facilitated by ROOTS and whether these 
critical variables explain variance in student mathematics 
outcomes within the treatment condition. Below, we define 
these three types of instructional interactions and briefly 
review the research that supports why they are particularly 
beneficial for at-risk learners and relevant for investigation 
within the ROOTS intervention groups. 

Overt Teacher Models 

Teacher models are intended to make new and complex 
mathematics content conspicuous to students (Archer & 
Hughes, 2011). Research suggests that teacher models are 
a more powerful way of presenting critical academic con- 
tent relative to discovery and problem-based teaching 
methods (Gersten et al., 2009; Mayer, 2004). Teacher 
models can entail step-by-step demonstrations of mathe- 
matical procedures and explanations of complex concepts. 
Think-alouds are another form of teacher modeling, where 
the teacher makes her mathematical thinking and prob- 
lem-solving processes overt to students. In ROOTS, for 
example, teachers might state, “I used my place value 
cards to show how 16 is made up of one ten and six ones.” 

Deliberate Student Practice Opportunities 

Research on mathematics interventions has found that 
deliberate practice opportunities are essential for improv- 
ing important mathematics outcomes (Gersten et al., 2009). 
In early mathematics, effective forms of student practice 
entail working with visual representations of mathematical 
ideas (e.g., place value blocks) and engaging in mathemati- 
cal discourse. Visual representations, when systematically 
included in instruction, allow students with MD the oppor- 
tunity to express mathematical ideas and transform the rep- 
resentations into abstract symbols (Gersten et al., 2009; 
NMAP, 2008). Student verbalizations are important 
because they permit students to convey their mathematical 
thinking and understanding of critical mathematics con- 
cepts before other modes of practice are instructionally 
appropriate, such as solving mathematics problems through 
written exercises (Doabler, Baker, et al., 2015). In ROOTS, 
tutors can prompt mathematics verbalizations from indi- 
viduals and groups of students. Individual response oppor- 
tunities permit tutors to monitor a student’s mathematical 
understanding and learning, whereas group responses 
allow more than one student the opportunity to verbalize 
their mathematical thinking. 
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Specific Academic Feedback 

Academic feedback refers to a tutor providing specific 
informational feedback about a student response or action. 
The ROOTS intervention offers teachers the strategies for 
how to provide timely corrective and affirmative feedback 
that is specific enough to extend learning opportunities, 
decrease the likelihood of later misconceptions, and address 
specific knowledge gaps. Research has demonstrated that 
teachers can increase mathematics achievement by provid- 
ing specific academic feedback (Gersten et al., 2009; 
Vaughn & Swanson, 2015). 

Taken together, these three types of instructional interac- 
tions (i.e., teacher models, student practice opportunities, and 
academic feedback) are of particular interest because they 
provide an ideal platform for exploring the “black box” of an 
empirically validated Tier 2 mathematics intervention. 
Examination of instructional interactions can also help ascer- 
tain how to intensify early mathematics interventions for stu- 
dents with MD and, in turn, provide the instructional 
experiences that lie at the heart of multitiered systems of sup- 
ports (MTSS) in mathematics and Response to Intervention 
(Rtl) service delivery models (D. Fuchs & Fuchs, 2015; L. S. 
Fuchs & Vaughn, 2012). For example, the benefit of certain 
behaviors, like practice opportunities, may have maximum 
effect on intensifying an intervention when delivered below a 
particular rate of occurrence. However, practice opportuni- 
ties may have limited value in intensifying an intervention 
after a certain threshold is reached. Moreover, greater impor- 
tance may lie with the frequency of explicit teacher models. 
We note that the role of these behaviors may interact with 
other factors such as the initial skill level of the group. For 
example, higher rates of practice opportunities for lower per- 
forming groups may be critical whereas they may have less 
relative value for higher performing groups. 

Purpose of the Study 

A growing research literature on early mathematics inter- 
ventions has begun to crystallize the importance of system- 
atic and explicit instruction for students with or at risk for 
MD (Gersten et al., 2009). However, to our knowledge, no 
intervention studies have examined whether rates of instruc- 
tional interactions offer explanatory power for detecting 
differences on outcomes within the treatment condition. 
This study sought to address this blank spot in the field of 
mathematics intervention research by examining whether 
instructional interactions within the ROOTS mathematics 
intervention contributed to the improvement of student 
mathematics achievement within the treatment condition. 
Two research questions were addressed in the study. 

Research Question 1: Does the rate of instructional 

interactions within the ROOTS intervention predict 


gains in student mathematics outcomes, net of mathe- 
matics achievement at the start of the year? 

Research Question 2: What is the relationship between 
student mathematics achievement at the start of kinder- 
garten and the subsequent rate of instructional interac- 
tions during the ROOTS intervention? 

Method 

Participants 

Forty-six ROOTS intervention groups from 14 kindergar- 
ten classrooms in Oregon and 32 kindergarten classrooms 
in Texas participated in the study. Each group represented 
one kindergarten classroom. Of the 46 classrooms, 33 
were located in public schools, eight in private schools, 
and five in charter public schools. All charter and private 
school classrooms were located in Texas. Public school 
classrooms were located in schools eligible for Title 1 
funding and had populations where an average of 76% 
qualified for free or reduced-price lunch programs. 

Each ROOTS group was taught by a school-employed 
paraprofessional. Of the 46 tutors, 11 held bachelor’s 
degrees and four had associate’s degrees. Twenty-three of 
the tutors held only high school diplomas. Eight of the 46 
tutors were certified in elementary education. The majority 
of the tutors were females (98%) and identified their race as 
White. No attrition among tutors occurred during the study. 

In the efficacy trial, the five lowest performing students 
in each of the 46 treatment classrooms were nominated by 
the teacher to receive the ROOTS intervention in addition 
to their core (Tier 1) mathematics instruction. The nomina- 
tion process entailed a three-step process. First, to be con- 
sidered eligible for the intervention, a student had to have a 
pretest score below the 40th percentile on the Test of Early 
Mathematics Ability-Third Edition (TEMA-3). From those 
students who qualified for intervention, teachers were pro- 
vided with student scores from a battery of curriculum- 
based measures that assessed students’ number proficiencies 
(see “Measures” section). Teachers then selected up to five 
students who demonstrated low performances on the num- 
ber sense measures. 

Of the 228 nominated ROOTS students, 77% were 
White, 8% Black, and 15% were missing race data. In addi- 
tion, 30% were English learners, 50% were male, and 58% 
identified their ethnicity as Hispanic. The average age of the 
ROOTS students at pretest was 5.5 years ( SD = 0.3). Nearly 
60% of the ROOTS students scored below the 10th percen- 
tile on a standardized mathematics measure in the fall of 
their kindergarten year. 

Research Design 

This study analyzed data collected during an IES-funded 
efficacy trial of the ROOTS kindergarten mathematics 
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intervention (Baker, Chard, Clarke, Smolkowski, & Fien, 
2008). The study took place in Oregon and Texas, respec- 
tively, during the 2009-2010 and 2010-2011 school years. 
Blocking on schools, the efficacy trial randomly assigned 
94 full-day kindergarten classrooms to either treatment ( n = 
46) or control (n = 48) conditions. Each of the 46 treatment 
classrooms provided one ROOTS intervention group. 
Because the purpose of this article was to explore the 
explanatory power of instructional interactions facilitated 
within the ROOTS condition, our analyses focused exclu- 
sively on observation and student performance data col- 
lected in the 46 ROOTS groups. 

Procedures 

Implementation of ROOTS. School-based tutors delivered 
the ROOTS intervention in small-group formats (5:1 stu- 
dent-tutor ratio), 3 days per week (20-min sessions) for 
approximately 18 weeks. All lessons were scripted to 
enhance implementation and retain the preciseness of the 
mathematical content. Each ROOTS lesson contained 
between four and five activities. The opening activity 
entailed the “Nifty Fifty” activity and its purpose was to 
promote students’ number identification skills and use of 
efficient counting strategies with a (1-50) number chart. 
Each Nifty Fifty activity corresponded to the number of 
lessons completed in the intervention program. For exam- 
ple, in Lesson 18, tutors used the Nifty Fifty activity to 
help children count and identify numbers up to 18. The 
opening activities also supported students’ conceptual 
understanding of and procedural fluency with rational 
counting (i.e., one-to-one correspondence) and identifying 
whether one group of objects is greater than, less than, or 
equal to another group of objects. The second activity typi- 
cally introduced a new mathematical concept or skill that 
was central to the lesson’s overall objective. For this activ- 
ity, tutors used concrete objects (e.g., counting blocks or 
number lines) to explicitly demonstrate and explain the tar- 
geted concept or skill. The third and fourth activities 
involved either continuous practice of the second activity 
or a review of previously learned material. The final activ- 
ity was a brief worksheet activity that tutors used to review 
the lesson’s content. Each worksheet included a “note 
home” (in both English and Spanish) to provide students 
with additional practice opportunities outside of school. 
Finally, to document individual student performance 
across ROOTS, tutors administered the intervention’s 
curriculum-embedded assessments every five lessons. 

ROOTS training. All tutors received three curriculum work- 
shops on intervention implementation and small-group man- 
agement techniques. Each 4-hr workshop offered tutors 
practice opportunities in using explicit teacher modeling, 
providing academic feedback, and facilitating multiple 


practice opportunities to struggling learners. The workshops 
also informed tutors that extended and more frequent instruc- 
tional interactions than those prescribed in the ROOTS pro- 
gram were permissible as long as the interactions (a) centered 
on the targeted mathematics content and (b) did not interfere 
with other aspects of implementation fidelity (e.g., complet- 
ing the lesson within 20 min). That is, when deemed neces- 
sary, tutors were allowed to go above and beyond the ROOTS 
program. For example, if students were struggling to grasp a 
particular mathematical concept, tutors were encouraged to 
redemonstrate the concept or provide additional practice 
opportunities to promote higher rates of student success. To 
further bolster implementation, in-class coaching was pro- 
vided during across the 18-week intervention. At the conclu- 
sion of each workshop and prior to heading out into the field, 
tutors had to complete a real-time checkout. This checkout 
had tutors implement a lesson activity and receive feedback 
on their delivery of instruction from research staff. Criteria 
were not specified for the training checkouts. 

Implementation fidelity. Implementation fidelity of ROOTS 
was assessed 3 times in each group across the 18-week 
intervention. The fidelity of implementation measure was 
specifically designed to target mathematics activities within 
each ROOTS lesson. Using a 3-point rating scale, which 
ranged from 0 (did not implement) to 0.5 (partial implemen- 
tation) to 1.0 (full implementation), observers coded 
whether tutors taught key design components described 
within each lesson activity. For example, tutors received a 
full implementation rating in the opening activity if they 
used the Nifty Fifty Chart and Quick Numeral cards. Fidel- 
ity scores were computed as the mean across all lesson 
activities. Clarke et al. (in press) reported moderate to high 
levels of implementation fidelity (M = 0.95, SD = 0.07). 

Measures 

Student measures. Students were assessed at pretest and 
posttest on a general outcome measure of students' proce- 
dural and conceptual knowledge of whole numbers, and a 
set of early mathematics curriculum-based measures that 
focused on discrete skills of number sense. Trained staff 
administered all student measures. 

The TEMA-3 (Pro-Ed, 2007) is a standardized, norm- 
referenced, individually administered measure of beginning 
mathematical ability (i.e., ages 3 to 8 years 11 months) with 
a population mean and standard deviation of 100 and 15, 
respectively. The TEMA-3 reports test-retest reliability 
data of .82 and .93 using alternative forms. For concurrent 
validity with other mathematics outcome measures, the 
TEMA-3 reports coefficients ranging from .54 to .91. 
Standard scores were used in subsequent analyses. 

Early Numeracy Curriculum-Based Measurement 
(EN-CBM; Clarke & Shinn, 2004) consists of four, 1-min 
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Within Group (Student Level) Model 



Between Group (Tutor Level) Model 



Figure 2. Path diagram of MLSEM. 

Note. MLSEM = multilevel structural equation model; TEMA = Test of Early Mathematics Ability. 


fluency-based measures: Oral Counting, Number 
Identification, Quantity Discrimination, and Missing 
Number. Clarke et al. (2016) report average test-retest reli- 
ability of EN-CBM as .89, and concurrent validity coeffi- 
cients between EN-CBM total scores and the TEMA-3 
scores at pretest (r = .87) and posttest ( r = .81). A total 
EN-CBM score was computed as the sum across all subtests 
and used in subsequent analyses. 

Observations of instructional interactions. Trained observers 
administered the Classroom Observations of Student- 
Teacher Interactions-Mathematics instrument (COSTI-M; 
Doabler, Baker, et al., 2015) to document the frequency of 
instructional interactions. The COSTI-M represents a modi- 
fied version of an observation system designed by 
Smolkowski and Gunn (2012). Observational data were 
collected during the 20-min ROOTS intervention sessions 
on the real-time occurrences of: (a) teacher models, (b) 
group responses, (c) individual responses, and (d) teacher- 
provided academic feedback. Incident rates of the instruc- 
tional interactions were calculated by dividing the frequency 
of instructional interactions in an observed lesson by the 
duration of the observation in minutes. 

By design, each ROOTS group was scheduled for observa- 
tion 3 times across the 18-week study, with roughly 5 weeks 


separating each observation occasion. Because of logistical 
difficulties in forming the groups, some ROOTS groups were 
observed 2 times. Due to scheduling errors, three groups were 
observed 4 times. Combined, 125 direct observations were 
conducted and on 29 of these occasions two observers col- 
lected data simultaneously to assess observer agreement. 
Intraclass correlation coefficients (ICCs) representing 
observer agreement of the instructional interactions ranged 
from .78 to .96, which suggests substantial to nearly perfect 
observer agreement (Smolkowski & Gunn, 2012). 

Statistical Analysis 

A series of multilevel structural equation models 
(MLSEMs), with students and repeated observations nested 
within tutors, tested both research questions separately for 
each instructional interaction. We estimated both of the key 
associations by (a) regressing latent instructional interac- 
tions on fall student mathematics achievement and (b) 
regressing fall to spring gains in student mathematics on fall 
student mathematics and latent instructional interaction. 
The basic MLSEM is shown as a path diagram in Figure 2. 

As depicted in Figure 2, a directional path is shown 
from fall achievement to the latent rates because fall 
achievement precedes the group observations in time. In 
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Table I. Descriptive Statistics for Student- and Group-Level Variables. 


Variables 

M (SD) 

Skewness 

Kurtosis 

Minimum 

Maximum 

n 

Student level 

TEMA-3 pretest 

77.25 (16.7) 

0.61 

-0.31 

55.00 

122.00 

205 

TEMA-3 posttest 

94.62 (13.3) 

-0.10 

0.47 

54.99 

141.00 

203 

TEMA-3 gains 

17.77(14.1) 

0.06 

1.01 

-38.00 

54.00 

186 

EN-CBM pretest 

40.24 (41.7) 

1.39 

1.36 

0.00 

196.00 

205 

EN-CBM posttest 

140.54 (49.7) 

-0.02 

-0.84 

26.00 

254.00 

203 

EN-CBM gains 

102.90 (42.7) 

0.29 

-0.22 

-18.00 

213.00 

186 

Group level 

TEMA-3 pretest 

76.95 (13.2) 

0.42 

-0.92 

55.40 

105.25 

46 

TEMA-3 posttest 

94.82 (10.5) 

0.08 

-0.30 

76.25 

1 19.67 

46 

TEMA-3 gains 

1 7.83 (9.8) 

1.07 

1.49 

1.50 

47.50 

46 

EN-CBM pretest 

39.30 (35.0) 

1.19 

0.45 

2.50 

1 34.00 

46 

EN-CBM posttest 

140.50 (38.6) 

-0.26 

-0.99 

66.67 

214.33 

46 

EN-CBM gains 

102.29 (30.8) 

0.48 

-0.43 

53.00 

178.75 

46 

Academic feedback 

1.18 (0.5) 

0.38 

-0.63 

0.16 

2.36 

46 

Individual responses 

1.67 (0.7) 

0.97 

0.34 

0.67 

3.58 

46 

Group responses 

1.36 (0.6) 

0.90 

0.63 

0.51 

3.18 

46 

Teacher models 

0.86 (0.3) 

0.96 

0.31 

0.30 

1.90 

46 

Note. Instructional interactions 

were averaged across 

the three observations 

occasions. TEMA-3 

= Test of Early Mathematics Ability-Third Edition; EN 

- 


CBM = Early Numeracy Curriculum-Based Measurement. 


turn, a directional path is shown between the latent rates 
and fall to spring achievement gains for the same reason. 
We used a latent variable approach for the rates to partial 
out time specific variation and random measurement error, 
correct for the anticipated modest reliability of the rates 
and thus get a more accurate estimate of effect size for the 
rates (Stoolmiller, Eddy, & Reid, 2000). Each occasion of 
observation served as an observed indicator of the underly- 
ing latent rate of interaction for a tutor across the interven- 
tion period. MLSEM was completed using Mplus (Muthen 
& Muthen, 2012) with the standard assumption of missing 
at random (MAR) for missing data and full information 
maximum likelihood estimation. All p values are two- 
tailed. Prior to and after model fitting, we performed exten- 
sive graphical inspection of univariate and bivariate 
distributions to identify potential violations of underlying 
model assumptions. 

Results 

Table 1 provides means, standard deviations, sample sizes, 
and skewness and kurtosis statistics for each student out- 
come, and a unit weighted composite rate score based on all 
the individual rates from each occasion of observation. 
Correlations between the composite rates of instructional 
interactions ranged from .25 to .58. The fall student out- 
comes both had mild floor effects but the fall to spring gain 
scores were more normally distributed. ICCs for the student 
outcomes were .53 and .62 for fall TEMA-3 and CBM, 
respectively; .52 and .45 for spring TEMA-3 and CBM, 


respectively; and .27 and .31 for TEMA-3 and CBM gains, 
respectively. The rate scores from each occasion of obser- 
vation did not suffer from floor effects (many zeros) but 
were somewhat positively skewed and so were log trans- 
formed to better approximate normality. 

Table 2 shows that across the eight models, overall 
model fit ranged from good to fair with RMSEA values 
ranging from .034 to .072, comparative fit index (CFI) val- 
ues ranging from .78 to .93, and Tucker-Lewis Index (TLI) 
values ranging from .77 to .92. ICCs for the three repeated 
observations of tutor rates were .10, .33, .25, and .33 for 
models, individual responses, group responses, and feed- 
back, respectively, and except for models, were significant 
at p < .05. The ICCs can be used to estimate the reliability 
of a composite based on the sum or average of the three 
observed rates using standard formulas and the reliabilities 
ranged from .25 to .60. These reliabilities are low enough to 
cause substantial bias in the estimation of the effects of the 
rates on subsequent achievement gains, which justifies the 
use of the latent variable approach for the rates. 

Tables 3 and 4 summarize the multilevel models. As 
shown in Table 3, fall mathematics achievement (TEMA-3 
and EN-CBM) was negatively and significantly correlated 
with only the latent rate of academic feedback. Latent rates 
were not significantly correlated with gains. In Table 4, for 
regressions, fall mathematics achievement significantly and 
negatively predicted the latent rate of academic feedback 
but all other regression effects involving the latent rate mea- 
sures were nonsignificant, including the regression of fall to 
spring gains on latent rates. 
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Table 2. Fit indices for MLSEMs. 







Rate (log transformed) 

Outcome 

x 2 

df 

p 

RMSEA 

CFI 

TLI 

Teacher models 

TEMA-3 

27.10 

15 

.028 

0.059 

0.783 

0.768 


EN-CBM 

28.25 

15 

.020 

0.062 

0.793 

0.779 

Individual responses 

TEMA-3 

22.87 

15 

.087 

0.048 

0.870 

0.862 


EN-CBM 

24.74 

15 

.054 

0.053 

0.857 

0.847 

Group responses 

TEMA-3 

19.02 

15 

.213 

0.034 

0.926 

0.921 


EN-CBM 

19.58 

15 

.189 

0.037 

0.925 

0.920 

Academic feedback 

TEMA-3 

24.13 

15 

.063 

0.052 

0.877 

0.869 


EN-CBM 

32.55 

15 

.005 

0.072 

0.798 

0.785 

Note. MLSEM = multilevel structural equation model; RMSEA = root mean 
Index; TEMA-3 = Test of Early Mathematics Ability-Third Edition; EN-CBM = 

square error approximation; CFI = comparative fit index; TLI 
Early Numeracy Curriculum-Based Measurement. 

= Tucker-Lewis 


Table 3. MLSEM Estimated Correlations. 

Variables 

Estimate 

SE 

z 

P 

Teacher models 





Log rate with initial TEMA-3 

0.26 

0.34 

0.77 

.43 

TEMA-3 gains with log rate 

0.06 

0.38 

0.16 

.86 

TEMA-3 gains with initial TEMA-3 

-0.62 

0.12 

-5.01 

.00 

Log rate with initial EN-CBM 

0.53 

0.35 

1.52 

.12 

EN-CBM gains with log rate 

-0.37 

0.38 

-0.99 

.31 

EN-CBM gains with initial EN-CBM 

-0.34 

0.17 

-1.97 

.04 

Individual responses 

Log rate with initial TEMA-3 

0.10 

0.21 

0.50 

.61 

TEMA-3 gains with log rate 

-0.10 

0.24 

-0.40 

.68 

TEMA-3 gains with initial TEMA-3 

-0.62 

0.12 

-5.01 

.00 

Log rate with initial EN-CBM 

0.08 

0.20 

0.42 

.67 

EN-CBM gains with log rate 

-0.18 

0.23 

-0.76 

.44 

EN-CBM gains with initial EN-CBM 

-0.34 

0.17 

-1.96 

.04 

Group responses 

Log rate with initial TEMA-3 

-0.40 

0.21 

-1.89 

.05 

TEMA-3 gains with log rate 

0.26 

0.26 

1.01 

.31 

TEMA-3 gains with initial TEMA-3 

-0.63 

0.12 

-5.04 

.00 

Log rate with initial EN-CBM 

-0.29 

0.21 

-1.38 

.16 

EN-CBM gains with log rate 

-0.23 

0.25 

-0.91 

.36 

EN-CBM gains with initial EN-CBM 

Academic feedback 

-0.34 

0.17 

-1.97 

.04 

Log rate with initial TEMA-3 

-0.71 

0.15 

-4.65 

.00 

TEMA-3 gains with log rate 

0.42 

0.23 

1.83 

.06 

TEMA-3 gains with initial TEMA-3 

-0.63 

0.12 

-5.06 

.00 

Log rate with initial EN-CBM 

-0.61 

0.15 

-3.85 

.00 

EN-CBM gains with log rate 

-0.06 

0.23 

-0.28 

.77 

EN-CBM gains with initial EN-CBM 

-0.35 

0.17 

-2.02 

.04 


Note. TEMA-3 = Test of Early Mathematics Ability-Third Edition; EN-CBM = Early Numeracy Curriculum-Based Measurement. 


To assess the robustness of our results, we reestimated 
our MLSEMs, including a binary indicator variable, to dis- 
tinguish public from other types of schools. The only signifi- 
cant effects obtained were on the initial math skill level 
(public schools substantially lower on both CBM and 
TEMA) and the rest of the results were quite similar to those 
shown in Tables 3 and 4. 


Discussion 

There is general consensus in the field that instructional 
interactions around critical mathematics content constitute 
the sinew of students’ development of early mathematics 
proficiency (Gersten et al, 2009; NMAP, 2008; NRC, 2001; 
Pianta & Hamre, 2009). However, little is known about the 
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Table 4. MLSEM Estimated Regressions. 


Variables 

Estimate 

SE 

Z 

P 

Teacher models 

Log rate on initial TEMA-3 

0.02 

0.03 

0.80 

.42 

TEMA-3 gains on log rate 

1.56 

2.50 

0.62 

.53 

TEMA-3 gains on initial TEMA-3 

-0.42 

0.12 

-3.40 

.00 

Log rate on initial EN-CBM 

0.02 

0.01 

1.73 

.08 

EN-CBM gains on log rate 

-5.31 

1 1.34 

-0.46 

.64 

EN-CBM gains on initial EN-CBM 

-0.14 

0.27 

-0.53 

.59 

Individual responses 

Log rate on initial TEMA-3 

0.02 

0.04 

0.501 

.61 

TEMA-3 gains on log rate 

-0.09 

0.58 

-0.169 

.86 

TEMA-3 gains on initial TEMA-3 

-0.38 

0.10 

-3.798 

.00 

Log rate on initial EN-CBM 

0.00 

0.01 

0.429 

.66 

EN-CBM gains on log rate 

-1.48 

2.28 

-0.648 

.51 

EN-CBM gains on initial EN-CBM 

-0.24 

0.13 

-1.752 

.08 

Group responses 

Log rate on initial TEMA-3 

-0.07 

0.04 

-1.772 

.07 

TEMA-3 gains on log rate 

0.06 

0.84 

0.072 

.94 

TEMA-3 gains on initial TEMA-3 

-0.38 

0.1 1 

-3.194 

.00 

Log rate on initial EN-CBM 

-0.01 

0.01 

-1.325 

.18 

EN-CBM gains on log rate 

-4.10 

3.19 

-1.285 

.19 

EN-CBM gains on initial EN-CBM 

-0.33 

0.15 

-2.191 

.02 

Academic feedback 

Log rate on initial TEMA-3 

-0.07 

0.01 

-3.746 

.00 

TEMA-3 gains on log rate 

-0.32 

2.04 

-0.156 

.87 

TEMA-3 gains on initial TEMA-3 

-0.40 

0.19 

-2.143 

.03 

Log rate on initial EN-CBM 

-0.02 

0.00 

-3.268 

.00 

EN-CBM gains on log rate 

-8.67 

6.56 

-1.323 

.18 

EN-CBM gains on initial EN-CBM 

-0.35 

0.17 

-2.026 

.04 


Note. MLSEM = multilevel structural equation model; TEMA-3 = Test of Early Mathematics Ability-Third Edition ; EN-CBM = Early Numeracy Curriculum- 
Based Measurement. 


explanatory power of instructional interactions within the 
treatment conditions of intervention studies. This study 
explored whether instructional interactions facilitated by a 
scripted intervention program might provide insight into 
whether or not differences in rates of instructional interac- 
tions explained variance in student mathematics outcomes 
within the ROOTS condition. 

Two research questions were addressed in the study. The 
first research question explored whether rates of teacher 
models, student practice opportunities, and academic feed- 
back within the treatment condition predicted meaningful 
improvements in the TEMA-3 and EN-CBM. We hypothe- 
sized that differences in the rate of systematic and explicit 
instructional interactions would predict gains in important 
mathematics outcomes. Results suggested that rates of 
teacher models, feedback, group responses, and individual 
responses within the intervention condition were not associ- 
ated with gains in the TEMA-3 and EN-CBM. 

It is essential to point out that our findings should not be 
construed as devaluing the critical importance of instmctional 


interactions when designing and delivering Tier 2 mathemat- 
ics interventions. The evidence is clear that these interactions 
are validated and necessary mechanisms of Tier 2 mathemat- 
ics interventions. However, the evidence is less clear about 
their role for contributing to student mathematics outcomes 
within intervention programs. A number of interesting ques- 
tions arise from these findings that warrant future 
consideration. 

Our ability to demonstrate predictive utility of the 
instructional interactions may have been affected by possi- 
ble threshold effects. Once instructional interactions reach a 
certain level or rate of delivery within soundly designed 
interventions such as ROOTS, they may fail to make a sub- 
stantively important impact on mathematics growth. Take 
the rate of individual response opportunities provided in the 
ROOTS groups as a possible example of this threshold 
effect. Results suggested that individual students received 
close to two opportunities per minute to verbalize or dem- 
onstrate their mathematical thinking. The benefit of mathe- 
matics verbalizations may be maximized below a rate of 
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two opportunities per minute and offer diminishing returns 
above that threshold. Thus, future research to address 
threshold effects is warranted. Additional research is also 
needed to examine other key aspects of instructional inter- 
actions such as timing or spacing (Rohrer & Pashler, 2010). 
For example, it may be that greater spacing of individual 
responses opportunities across the 20-min ROOTS lessons 
has a more pronounced effect on student mathematics out- 
comes than when the total number of opportunities exceeds 
a specified threshold of mathematics practice. 

The quality of instructional interactions may have also 
influenced the predictive utility of the instructional interac- 
tions. Prior observational research suggests that student 
mathematics achievement is related to the quality of instruc- 
tional interactions provided in mathematics instruction 
(Doabler, Baker, et al., 2015; Pianta & Flamre, 2009). 
However, a serious limitation or challenge may lie with the 
COSTI-M observation instrument and its theoretical prin- 
ciples. The COSTI-M is limited to capturing the frequency 
of observable instructional interactions during real-time 
instruction and thus was unable to document whether the 
interactions that occurred during the ROOTS intervention 
were of high or low quality. Other observation tactics, such 
as technology-based systems may have the capacity to 
enable deeper investigation of instructional interactions 
(Connor, 2013). For example, researchers could use tablets 
to simultaneously code the frequency, quality, duration, and 
cognitive demand of instructional interactions. 

Our second research question explored the relationship 
between the rate of instructional interactions and student 
mathematics achievement at the start of kindergarten. Our 
findings revealed that rates of teacher models, and group 
and individual responses were not associated with students’ 
pretest TEMA-3 and EN-CBM scores. Academic feedback, 
however, was found to have a negative and significant rela- 
tionship with fall mathematics achievement. Findings indi- 
cated that groups with lower student mathematics 
achievement at the start of kindergarten received higher 
rates of academic feedback. In the context of small-group 
instruction, academic feedback, both corrective and affir- 
mative, is an optimal platform for teachers to differentiate 
learning opportunities for students with MD. When teachers 
provide timely, specific academic feedback they can address 
misconceptions of mathematics tasks and build upon stu- 
dents’ current mathematics thinking and understanding 
(Gersten et al., 2009). 

Notwithstanding the importance of academic feedback 
for students with MD, questions remain about why tutors of 
the lower performing groups facilitated more academic 
feedback. One explanation is that, while kept blind to stu- 
dents’ TEMA-3 and EN-CBM pretest scores, these tutors 
made “local adaptations” (Forman et al., 2013) to ROOTS 
to best meet the instructional needs of their students. Tutors 
of the lower performing groups may have used ROOTS’ 


curriculum-embedded assessments to gauge student perfor- 
mance and subsequently provide additional academic feed- 
back than otherwise prescribed within the intervention. 
However, we are unable to determine whether student per- 
formance on the curriculum-embedded assessments was the 
driving factor behind the instructional adjustments in the 
lower performing groups. Future studies are therefore 
needed to understand how and why tutors might locally 
adapt ROOTS, and whether such adaptations are beneficial 
to students. 

Because a fine line exists between program fidelity and 
program adaptation, deeper investigations are also war- 
ranted to distinguish the “essential” features of Tier 2 inter- 
ventions that are adaptable from ones that must be delivered 
as designed (Forman et al., 2013). Essential features of the 
ROOTS intervention that tutors can adjust to provide better 
local fit include the provision of instructional interactions. 
For example, ROOTS tutors are encouraged to offer addi- 
tional academic feedback to provide greater clarity and 
specificity of complex mathematics concepts and proce- 
dures. However, as emphasized in the ROOTS workshops, 
such local adaptation was permissible if it remained in the 
boundaries of implementation fidelity (e.g., completing 
each lesson within 20 min). Conversely, the mathematics 
content targeted within ROOTS is not flexible or adaptable 
as tutors are required to teach the prescribed topics. 

Limitations and Implications for Research 

A number of limitations must be considered when interpret- 
ing our results. One limitation is the study’s sample size. 
Data from only 46 intervention groups were analyzed in the 
study. A larger sample size will likely be required to detect 
small yet meaningful group differences that may exist within 
a highly specified treatment condition. Relatedly, missing 
observation data is another possible limitation. Across the 
intervention, 16 observation occasions were missed. 

Another possible limitation is that ROOTS groups were 
scheduled for only three observations. Although scheduling 
three observations is common among observational research 
studies (e.g., Pianta & Hamre, 2009; Smolkowski & Gunn, 
2012), conducting four or more observations per instruc- 
tional group may provide an unequivocal perspective of 
instructional interactions and the ways in which they can be 
manipulated to increase intervention intensity (D. Fuchs & 
Fuchs, 2015; Yoder & Woynaroski, 2015). Our limit of 
three observation occasions was primarily driven by 
resource constraints in the larger efficacy trial. 

Finally, although our work has focused extensively on 
observed instructional interactions, teachers’ mathematics 
knowledge might offer an additional way to examine varia- 
tions in treatment outcomes (Ball, Hill, & Bass, 2005). It 
may be that teachers with in-depth understanding of math- 
ematics may offer a potentially more intensive experience 
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for struggling learners than from teachers who are less 
skilled in mathematics even though the teachers are utiliz- 
ing the same instructional materials. For example, a teacher 
with extensive knowledge of mathematics may be able to 
represent whole number concepts in different ways than 
otherwise prescribed in mathematics interventions. Teachers 
who can draw on their mathematical knowledge are also 
likely to be more effective at linking an intervention with 
students’ prior understandings of mathematics. Future 
research is warranted to investigate the role teachers’ con- 
tent knowledge may play in the variance of student out- 
comes within Tier 2 interventions. 

Conclusion 

There is an urgent need not only to develop effective math- 
ematics interventions but also to systematically study the 
underlying mechanisms of those interventions that acceler- 
ate student achievement. Although using direct observa- 
tions to document important instructional interactions can 
be challenging (Connor, 2013), this measurement tactic has 
strong potential to further unpack the black box of mathe- 
matics interventions and evaluate the extent to which stu- 
dents at risk for MD receive intensive mathematics 
instruction. Doing so will help to ensure that all learners 
maximize their potential in understanding mathematics. 
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